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Abstract.  The  assessment  and  treatment  of  traumatic  brain  injury  (TBI)  has 
become  a  difficult  challenge  for  the  DoD  medical  health  system.  Clinical 
neuropsychologists  are  being  asked  to  make  statements  regarding  a  soldier’s 
functional  skills,  ability  to  return  to  active  duty,  and  competence  in  tasks  of 
community  living.  Given  the  increasing  prevalence  of  blast  injuries  to  the  head, 
and  the  fact  that  many  brain  injuries  may  have  no  external  marker  of  injury,  there 
is  need  for  researching  innovative  assessment  methods  in  detecting  blast-related 
brain  injury.  To  address  these  issues,  two  virtual  reality-based  Paced 
Auditory/Visual  Serial  Addition  Tests  (PA/VS AT)  were  developed  that  involve 
the  participant  being  immersed  in  a  Virtual  Middle  Eastern  City  as  serial  addition 
stimuli  are  presented.  This  study  is  an  initial  validation  of  the  VRPASAT  and 
VRPVSAT  as  assessments  of  neurocognitive  functioning.  When  compared  to  the 
paper- and-pencil  version  of  the  test,  as  well  as  the  Automated  Neuropsychological 
Assessment  Metrics,  the  VRPASAT  and  VRPVSAT  appear  to  have  enhanced 
capacity  for  providing  an  indication  of  a  participant’s  performance  while 
immersed  in  a  military  relevant  simulation. 

Keywords.  Virtual  Reality,  Paced  Serial  Assessment  Test,  Neuropsychological 
Assessment,  Military. 


Introduction 

Military  service  members  deployed  in  support  of  Operation  Iraqi  Freedom  and 
Operation  Enduring  Freedom  are  at  increased  risk  of  traumatic  brain  injuries  (TBIs). 
The  RAND  Corporation  reported  that  nearly  one  in  five  service  members  who 
deployed  to  Iraq  or  Afghanistan  reported  a  probable  TBI  [1].  Further,  12-20%  of 
Service  Members  report  symptoms  of  TBI  during  re-deployment  [2].  Blast  injuries 
often  produce  symptoms  similar  to  classical  TBI,  thereby  complicating  detection, 
diagnosis,  and  treatment  [3].  Military  service  members  with  TBI  experience  problems 
in  efficiently  and  successfully  completing  work  tasks  and  some  find  it  difficult  to  even 
return  to  work  [4]. 

The  neuropsychological  examination  is  one  of  the  methods  used  by  military 
neuropsychologists  to  diagnose  acquired  disorders  of  brain  function.  One  of  the  most 
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widely  used  assessments  of  TBI  is  the  Paced  Auditory  Serial  Addition  Test  (PASAT) 
[5].  The  PASAT  requires  the  patient  to  successfully  complete  a  variety  of  cognitive 
functions:  sustained  attention,  working  memory,  and  the  simultaneous  time-constrained 
performance  of  multiple  neurocognitive  tasks  [6].  There  is  also  a  Paced  Visual  Serial 
Addition  Test  (PVSAT)  that  presents  the  digits  visually  on  a  computer  screen  [7].  A 
number  of  studies  with  various  patient  groups  (e.g.,  TBI;  and  multiple  sclerosis)  have 
found  PVSAT  performance  to  be  superior  to  PASAT  performance  [8].  Additionally, 
there  are  computerized  neuropsychological  assessments  that  have  been  developed  to 
automate  the  paper  and  pencil  assessments.  For  example,  the  Army  gives  the 
Automated  Neuorpsychological  Assessment  Metrics  (ANAM)  battery  to  all  soldiers 
prior  to  deployment. 

Although  tests  like  the  PASAT  and  ANAM  are  widely  used,  providers  in  military 
contexts  have  increasingly  been  asked  to  use  neuropsychological  test  performances  to 
make  recommendations  about  patients’  everyday  functioning  [9].  Clinicians  working  in 
the  deployed  environment  or  at  military  treatment  facilities  may  use  cognitive 
assessments  to  inform  questions  related  to  fitness  for  duty.  For  example,  deployed 
commanders  may  have  referral  questions  related  to  the  safety  of  personnel  to  perform 
basic  tactical  skills.  On  the  home  front,  military  neuropsychologists  may  be  consulted 
as  part  of  a  “fitness  for  duty”  evaluation  that  is  conducted  when  impairments 
significantly  interfere  with  work  performance.  In  addition,  there  is  increasing  interest  in 
the  assessment  of  the  severity  of  functional  impairment  following  TBI. 

The  complexity  and  lethality  of  modern  warfare  place  great  demands  on  a  service 
member’s  neurocognitive  resources.  At  varying  levels  of  threat,  service  members  must 
be  able  to  exercise  control  of  cognitive  functions.  It  may  be  challenging  to  interpret  the 
results  of  traditional  cognitive  assessment  tools  to  answer  military  specific  questions. 
With  tremendous  individual  variability  in  responses  to  stress,  how  well  does 
performance  during  a  well-controlled  cognitive  assessment  predict  performance  during 
the  stresses  of  war?  Following  a  mild  TBI,  how  do  we  assess  the  functional  impairment 
of  service  members  whose  occupational  environment  has  significant,  unpredictable  low 
and  high  intensity  stress?  Hence,  for  a  measure  to  be  relevant  to  an  assessment  of  a 
service  member’s  neurocognitive  functioning,  it  should  provide  some  indication  of  a 
service  member’s  cognitive  performance  within  high  and  low  threat  settings. 

Developments  in  the  area  of  virtual  reality  may  offer  new  opportunities  to 
improve  ecological  validity  and  inform  key  questions  related  to  the  post-TBI 
assessment  of  service  members.  Virtual  environments  (VEs)  allow  for  creation  of 
simulated  realistic  environments  in  which  performance  can  be  tested  and  trained  in 
systematic  fashion.  By  designing  VEs  combined  with  allied  computer-based 
assessment  tests  we  can  create  “virtual  assessment  environments”  that  incorporate 
challenges  based  on  functional  behaviors  bringing  greater  ecological  validity  to 
assessment  and  rehabilitation  methods.  Within  such  a  VE,  the  experimental  control 
required  for  rigorous  scientific  analysis  and  replication  can  still  be  maintained  within 
simulated  contexts  that  embody  the  complex  challenges  found  in  naturalistic  settings. 


1.  Methods  and  Materials 

This  study  was  designed  as  an  initial  validation  of  the  VRPASAT  and  VRPVSAT  in  a 
military  cohort.  The  University  of  Southern  California  and  a  large  Army  installation  in 
the  continental  U.S.  are  collaborating  on  an  initial  pilot  study  to  validate  the 
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VRPASAT  and  VRPVSAT  with  active  duty  military  personnel.  We  compared  paper- 
and-pencil  PASAT,  ANAM,  VRPASAT,  and  VRPVSAT  on  behavioral  measures.  Our 
primary  hypotheses  were:  1)  the  VRPASAT  (but  not  the  VRPVSAT)  would  be 
significantly  correlated  with  the  paper-and  pencil  PASAT;  and  2)  the  VRPASAT  and 
VRPVSAT  would  be  correlated  with  ANAM  subtests  assessing  mathematical 
processing  and  procedural  reaction  time. 

1.1.  Participants  and  Procedure 

The  University  of  Southern  California’s  Institutional  Review  Board  approved  the  study. 
A  total  of  49  (94%  were  male)  military  service  members  participated  in  the  study.  Fifty 
percent  of  participants  had  3  years  or  less  of  military  service  experience,  and  years  of 
experience  ranged  from  1  to  24  years.  All  participants  were  fluent  English  speakers  and 
had  normal  or  corrected  to  normal  vision.  Each  participant  gave  informed  consent  prior 
to  participation  in  this  research  study.  After  informed  consent  was  obtained,  basic 
demographic  information  was  recorded.  Next,  participants  were  assessed  on  the 
PASAT;  ANAM,  VRPASAT,  and  VRPVSAT. 

Paced  Auditory  Serial  Assessment  Test:  The  Paced  Auditory  Serial  Addition  Test 
(PASAT)  assesses  auditory  information  processing  speed  and  flexibility,  as  well  as 
mathematical  calculation  ability.  The  PASAT  is  auditorially  presented  to  control  the 
rate  of  stimulus  presentation.  Each  participant  was  presented  with  a  series  of  single 
digit  numbers,  in  which  the  two  most  recent  digits  must  be  summed.  For  example,  if 
the  digits  “2”,  “6”  and  “3”  were  presented,  the  participant  was  to  respond  with  the 
correct  sums:  “8”  and  then  “9”.  Practice  trials  occurred  prior  to  the  first  trial.  Stimuli 
are  presented  at  four  different  rates  (2.4  s,  2.0  s,  1.6  s,  and  1.2  s).  The  duration  of  each 
digit  was  approximately  .4  s. 

Automated  Neuropsychological  Assessment  Metrics:  The  Automated 
Neuropsychological  Assessment  Metrics  (ANAM)  is  a  psychometrically  validated  tool 
designed  to  detect  speed  and  accuracy  of  attention,  memory,  and  executive  function 
[10].  ANAM  software  includes  a  timing  mechanism  to  insure  accuracy  across 
computer  platforms  and  supports  immediate  analysis  and  report  generation.  The 
ANAM  records  an  individual's  performance  through  a  series  of  responses  to  computer- 
based  prompts.  ANAM  has  been  widely  used  by  the  U.S.  Department  of  Defense  and 
allows  for  serial  assessment  [11].  The  following  ANAM  subtests  were  given  in  the 
order  presented:  TBI  Questionnaire;  Sleepiness  Scale;  Mood  Scale;  Simple  Reaction 
Time;  Code  Substitution;  Procedural  Reaction  Time;  Math  Processing;  Matching  to 
Sample;  and  Go/No-Go. 

Virtual  Reality  for  Cognitive  Performance  and  Adaptive  Training:  At  the 
University  of  Southern  California’s  Institute  for  Creative  Technologies,  we  have 
developed  an  adaptive  virtual  environment  for  assessment  and  rehabilitation  of 
neurocognitive  and  affective  functioning.  This  project  brings  together  a  team  of 
researchers  to  incorporate  cutting  edge  neuropsychological  and  psychophysiological 
assessment  into  state  of  the  art  interactive/adaptive  virtual  Iraqi/Afghani  scenarios 
(City,  Checkpoint). 

Two  primary  goals  define  these  virtual  and  adaptive  environments:  1)  a  Virtual 
Reality  Cognitive  Performance  Assessment  Test  (VRCPAT  1.0)  that  includes  a 
normative  database  drawn  from  a  battery  of  neuropsychological  and 
psychophysiological  measures  for  diagnostic  assessment  and  treatment  of  Soldiers  with 
affective  disorders,  brain  injury,  or  neurocognitive  deficits;  and  2)  a  Virtual  Reality  for 
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Cognitive  Performance  and  Adaptive  Treatment  (VRCPAT  2.0)  that  develops  an 
adaptive  environment,  in  which  data  gleaned  from  the  assessment  module  (VRCPAT 
1.0)  is  used  for  refined  analysis,  management,  and  rehabilitation  of  Soldiers  who  have 
suffered  blast  injuries  (varying  levels  of  traumatic  brain  injury)  and/or  are  experiencing 
combat  stress  symptoms  [12].  For  the  current  study,  we  focused  on  results  from  1)  a 
Middle  Eastern  City;  and  2)  a  checkpoint  in  a  Middle  Eastern  City.  For  the  VRPASAT 
(a  virtual  reality  variation  of  the  traditional  PASAT),  we  assessed  soldiers  while  they 
virtually  ambulated  through  a  Virtual  Middle  Eastern  City.  After  following  a  fire  team 
to  6  zones,  the  participant  was  tested  on  navigation  and  spatial  memory  by  being 
instructed  to  turn  around  and  find  the  way  back  through  each  of  the  zones  in  reverse 
order  (see  Figure  1). 


Figure  1.  Middle  Eastern  City. 

For  the  VR  version  of  the  Paced  Visual  Serial  Addition  Test  (VRPVSAT),  soldiers 
experienced  a  checkpoint  in  a  Middle  Eastern  City.  An  area  of  the  Middle  Eastern  City 
has  been  constructed  to  resemble  a  traffic  checkpoint  with  a  variety  of  moving  vehicles 
arriving,  stopping  and  then  moving  onward  into  the  city  (see  Figure  2). 


Figure  2.  Middle  Eastern  Checkpoint. 
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Participants  listened  to  a  virtual  trainee  as  the  trainee  classified  passing  vehicles.  In 
between  vehicle  presentations,  the  user  was  exposed  to  visual  PASAT  stimuli.  Herein 
the  numbers  are  presented  against  an  Iraqi  checkpoint  background  [7], 

1.2.  Data  Analytics 

Correlations  were  run  between  the  three  types  of  PASAT,  and  the  virtual  reality  based 
PASATs  were  also  correlated  with  various  tests  from  the  ANAM.  In  particular,  we 
were  interested  in  the  relationship  of  the  participants’  performance  on  the  PASATs  and 
the  mathematical  processing  and  procedural  reaction  time  tests. 

A  separate  set  of  analyses  were  conducted  to  assess  differences  in  participants’ 
performance  on  the  various  PASATs  employed.  A  3  (PASAT  type)  by  2  (session) 
repeated  measures  ANOVA  was  utilized  to  serve  this  purpose.  Separate  ANOVAs 
were  employed  to  test  differences  between  the  virtual  reality  PASATs  in  comparison  to 
the  paper  and  pencil  version  during  the  first  two  sessions,  and  the  second  two  sessions. 


2.  Results 

Correlations  between  performance  on  the  VRPVSAT  and  tests  in  the  ANAM  test 
battery  revealed  that  VRPVSAT  performance  is  correlated  with  mathematical 
processing,  r  =  0.50,  p  <  0.001,  as  well  as  performance  on  the  procedural  reaction  time 
test,  r  =  0.41,  p  <  0.01.  Likewise,  the  VRPASAT  was  also  correlated  with  both 
mathematical  processing,  r  =  0.43,  p  <  0.01,  and  procedural  reaction  time  performance, 
r  =  0.47,  p<  0.01. 

The  correlation  analysis  involving  all  three  PASATs  revealed  that  the  VRPASAT 
was  correlated  with  the  paper  and  pencil  version  of  the  PASAT,  r  ~=  0.78,  p  <  0.001. 
The  VRPVSAT  and  the  paper  and  pencil  PASAT  were  not  significantly  correlated,  r  = 
0.26,  p  =  0.07. 

The  ANOVA  involving  all  three  PASATs  and  using  the  first  two  sessions  of  the  paper 
and  pencil  PASAT  (2.4  and  2.0  respectively),  evidenced  a  main  effect  of  PASAT  type, 
F(2,  47)  =  72.54,  p  <  0.001,  such  that  the  highest  performance  was  found  during  the 
VRPVSAT  as  a  paired-samples  t-tests  revealed  that  performance  was  significantly 
higher  than  performance  on  both  the  VRPASAT,  t(48)  =  11.60,  p  <  0.001,  and  the 
paper  and  pencil  PASAT,  t(48)  =  3.43,  p  <  0.01.  The  VRPASAT  was  the  most  difficult 
task  as  performance  was  significantly  higher  on  the  paper  and  pencil  PASAT,  t(48)  = 
11.12,  p  <  0.001.  A  session  main  effect  was  also  revealed,  as  performance  on  the  first 
session  was  higher  in  general,  F(l,  48)  =  39.84,  p  <  0.001.  The  interaction  between 
PASAT  type  and  session  was  also  significant,  F(2,  47)  =  10.76,  p  <  0.001,  due  to  the 
lack  of  a  significant  difference  in  performance  from  session  1  to  session  2  in  the 
VRPVSAT  (see  Figure  3). 
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PASAT  Comparison  with  Paper  and  Pencil 
Sessions  1  and  2 
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Figure  3.  PASAT  Comparisons  (Sessions  1  &  2) 

The  ANOVA  utilizing  the  third  and  fourth  sessions  of  the  paper  and  pencil  PASAT 
(1.6  and  1.2  s  respectively)  evidenced  the  same  main  effect  and  interaction  results. 
However,  the  difference  between  performance  on  the  paper  and  pencil  and  VRPASAT 
was  no  longer  significant,  t(48)  =  0.452,  p  =  0.65  (see  Figure  4). 


PASAT  Comparison  with  Paper  and  Pencil 
Sessions  3  and  4 
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Figure  4.  PASAT  Comparisons  (Sessions  3  &  4) 


3.  Discussion 

The  Checkpoint  VRPVSAT  and  the  City  Walkthrough  VRPASAT  were  correlated 
with  both  mathematical  processing  and  procedural  reaction  time.  These  results  are 
consistent  with  the  view  that  the  PASAT  measures  speed  of  information  processing; 
mathematical  ability,  and  attention  [8]. 

Participant  performance  on  the  City  Walkthrough  VRPASAT  had  a  strong  positive 
correlation  with  performance  on  the  paper  and  pencil  PASAT,  demonstrating  high 
levels  of  convergent  validity  between  the  two  tests.  Differences  in  performance 
between  the  two  PASATs  were  evidenced  during  the  first  two  sessions  of  the  paper  and 
pencil  PASAT,  but  not  during  the  second  two,  more  difficult,  sessions.  This  result  may 
be  explained  by  the  fact  that  the  City  Walkthrough  VRPASAT  includes  the  secondary 
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memory  task  as  well  as  the  added  distracting  environmental  stimuli.  Thus,  even  when 
the  interstimulus  interval  involved  is  longer  during  the  virtual  reality  based  PASAT,  the 
task  is  more  difficult  due  to  other  stimuli  requiring  cognitive  resources.  The  City 
Walkthrough  VRPASAT  may  also  allow  for  more  variability  in  performance  without 
adding  to  participant  frustration  by  speeding  up  the  test. 

As  hypothesized,  the  VRPVSAT  and  the  paper  and  pencil  PASAT  were  not 
significantly  correlated.  These  results  are  consistent  with  the  studies  using  various 
patient  groups  (e.g.,  TBI;  and  multiple  sclerosis)  that  have  found  PVSAT  performance 
to  be  superior  to  PASAT  performance  [8].  This  superior  visual  performance  is  typically 
interpreted  as  representing  an  “interference”  framework,  in  which  the  articulation  of  the 
response  aurally  interferes  with  the  aural  presentation  of  the  digits.  Hence,  less 
interference  exists  between  incoming  visual  stimuli  and  the  articulated  response  on  the 
VRPVSAT  than  on  the  VRPASAT.  As  a  result,  VRPVSAT  may  proffer  a  more  robust 
measure  of  speed  of  information  processing. 

Future  work  should  assess  the  clinical  validity  of  the  VRPASAT  and  VRPVSAT 
in  a  clinical  population.  Although  further  validation  with  an  intact  civilian  population  is 
needed,  assessment  of  the  difference  between  the  VRPASAT  and  VRPVSAT  may  have 
important  clinical  implications. 
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